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ABSTRACT 

The General Accounting Office (GAO) examined the 
long-term earnings and employment outcomes of Job Training 
Partnership Act (JTPA) programs. Data from the National JTPA Study 
and annual earnings records from the Social Security Administration 
were used to calculate the average earnings and employment rates of 
four target groups (adult men, adult women, male youths, and female 
youths) for 5 years after their acceptance into the study. Although 
the statistical analysis showed some positive effects of JTPA in the 
years immediately following training, JTPA failed to produce any 
significant effect on earnings or employment rates after 5 years. In 
some earlier years, adult men and women who received training had 
earnings or employment rates significantly higher than did their 
counterparts in the control group. By year 5, however, each of the 
four treatment groups had earnings and employment rates that were 
nominally higher than those of the control group. The higher earnings 
could not be attributed to JTPA rather than to chance alone, however, 
because none of the fifth-year differences was s tat i s t i cal ly 
significant. (Twenty tables/figures are included. Appended are the 
following: data sources and methodology; data and statistics for the 
report’s figures; and comments from the Department of Labor and GAO.) 
(MN) 
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GAO 



United States 

General Accounting Office 

Washington, D.C. 20548 



Health, Education, and 
Human Services Division 

B-256749 

March 4, 1996 

The Honorable Paul Simon 
United States Senate 

Dear Senator Simon; 

Although the federal government spends billions of dollars annually to 
support employment training programs, little is known about their 
long-term effects on participants’ earnings and employment rates. ^ Few 
training programs have been rigorously evaluated to assess their net 
impact, and, for those that have, the research results have often been 
inconclusive. Furthermore, most research on training programs has 
focused on short-term effects — the year or two immediately following 
completion of training. 

Enacted in 1982, title II of the Job Training Partnership Act (jtpa) has been 
the cornerstone of federal employment training programs, providing block 
grants to state and local governments to administer these federally funded 
programs, jtpa supports job training for individuals facing barriers to 
employment and needing special training to obtain productive 
employment. Under recent legislative proposals^ to consolidate multiple 
federally funded training programs, states would have the flexibility to 
design and implement a statewide approach to job training based on the 
concept of one-stop career centers. As states design and implement their 
approaches to job tiaining, lessons learned from jtpa can help in 
reallocating training dollars and in setting performance standards. 

Because of concerns about the long-term impact of job training programs, 
you asked us to (1) ascertain the long-term earnings of participants in 
jTPA-sponsored programs and to compare their earnings with those of 
nonparticipants and (2) calculate the long-term employment rates for 
these .ITPA participants and to compare their employment rates with those 
of nonparticipants. For this study, we defined long-term earnings and 
employment rates as the annual earnings and employment rates achieved 
in the fifth year after applying for jtpa training. 



'M ultiple Km ploymon t Tr aining Programs: Mo st Federal Agencies Do Not Know If Their IVograms Are 
Working Effectively (GAO/llEHS-94-88r Mar. 2. 1994). ' ' ' 



“Both the .Senate imd tlie Ii(Uis(‘ liave pas.sed bills that would consolidate over !K) ferieral education, 
employment, and job training programs. Tlie Senate's Workforce Development Act of 1995 (S. 

14.'5) would replace the prognuns with one block grant to each state, while the House’s Consolidated 
and Reformed Education, Employment, and Rehabilitation Systems Act (CAREERS Act, H.R. 

1617) would authorize three consolidation grants to each state. 
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To develop this information, we merged data from the National .itpa Study^ 
(njs) with annual earnings records from the Social Security 
Administration (ssa). Participants in the njs were randomly assigned to 
either the treatment group (allowed to enroll in jtpa training) or the 
control group (not allowed to enroll in jtpa training for 18 months). We 
calculated the averse earnings and employment rates'’ of four target 
groups (adult men, adult women, male youths, and female youths'^) for 5 
years after their acceptance into the study. We considered a 5-year period 
sufficient to provide meaningful data on the long-term effects of jtpa 
training. 

Because of the njs’ inherent design problems, we cannot unequivocally use 
our findings to draw conclusions about jtpa’s effectiveness. For example, 
the participating areas were neither randomly selected nor necessarily 
representative of jtpa training nationally. Moreover, not everyone assigned 
to the treatment group enrolled in or completed jtpa training. Also, many 
control group members received some training services from other 
sources. Nevertheless, the njs had some design advantages, such as 
random assignment of applicants for jtpa training, that made the data 
appropriate for our study. 

Appendix I has further information on the njs and ssa data sets and our 
statistical results. We conducted our work between April 1994 and 
January 1996 in accordance with generally accepted government 
accounting standards. 



Although our statistical analysis showed some positive effects of .rrPA in 
the years immediately following training, we found no significant® effect of 
JTPA on earnings or employment rates after 5 years. In some earlier years, 
adult men and women who received training — but not male or female 



^The Department of Labor commissioned Abt Associates to conduct the Natioral JTl^A Study in 1986 
to evaluate the impact of JTPA on adults and youths. 

"^We defined employment on the basis of SSA earnings records. If a jx'rsrms SSA earnings record 
showed positive earnings in a given calendar year, we considere<l that person to be employed. If 78 out 
of 100 people in a group showed positive eaniings in their SSA earnings records, the employment rate 
for that group would be 78 percent. 

^’Adults were defimxi as those 22 years old and older. Youths were defined as thos<* agt^d 16 to 21 w'ho 
no longer attended school when applying for JTPA training. This includtHl both high school dropouts 
and graduates. 

‘"“Significauco’’ refers to statist ii'al significance at the 5-percent level. Thissignifu’ance luetuisthat we 
can be 05-i>ercent (’onfident that the obs(‘rv('d difforxmce between groups is not due t(.» chamre or 
random variation. Our convention is to calculate significance at the 5-pereent level. We list* this 
convention both in reference to other reports imd in presenting our own findings. 
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youths— ha I earnings or employment rates significantly higher than those 
of the control group. By the fifth year, each of the four treatment groups 
had earnings and employment rates that were nominally higher than those 
of the contrcJ group. Because none of the fifth-year differences were 
statistically significant, however, we could not attribute the higher 
earnings to jtpa training rather than to chance alone. 



Enacted in 1982, jtpa is the largest federal emplo 5 unent training program, 
with titles II-A and II-C intended to prepare economically disadvantaged 
adults and youths, respectively, for entry into the labor force.'^ jtpa 
emphasizes state and local government responsibility for administering 
federally funded job training programs. In fiscal year 1995, jtpa title II-A 
and II-C program.s received ^proximately $1.6 billion in funding. 

•ITPA training programs fmnually provide employment training for specific 
occupations and services, such as job search assistance and remedial 
education, to roughly one million economically disadvantaged individuals. 
Training is provided in local service delivery areas (sda) through service 
providers, such as vocational-technical high schools, community colleges, 
proprietary schools, and community-based organizations. The program 
objectives ?re to increase earnings and employment and to reduce welfare 
dependence for participants of all ages. During the njs, participation in 
JTPA involved roughly 3 to 4 months of training at an average cost of about 
$2,400 per participant. 

In 1986, Labor commissioned the njs to evaluate the impact of jtpa on 
adults and youths because previous findings on the effects of job training 
programs had been hampered by poor data and statistical problems. The 
NJS randomly assigned persons who sought jtpa services, and were eligible 
for them, to a treatment group or a control group. The treatment group 
was offered jtpa training, and the control group was not. The study was 
intended to ensure that the two groups would not differ systematically in 
any way except access to the program, so any subsequent differences in 
outcomes could be attributed solely to jtpa. 

The study included over 20,000 eligible participants who aqjplied for jti^a 
services between November 1987 and September 1989 in 16 local sdas. The 



ntTo n to Ixjth adulta and youtlis, Tho 1992 JTPA Atm-ndmcnUs .split 

titio 11-A into two component.s: titU> 11-A for adults and titio 11-C for youths. ' 
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Study followed up on a sample of people in the treatment and control 
groups 18 months after assignment® and then again at 30 months.® 

The NJS showed mixed results on the impact of jtpa programs.^® Adult 
women assigned to jtpa training had significantly higher earnings than the 
control group of adult women after 18 and 30 months, but the treatment 
group of adult men, as well as of both male and female youths, did not 
have significantly higher earnings than its respective control groups. 



Participant Earnings 
Not Significantly 
Greater Than Control 
Group Earnings After 
5 Years 



Participants assigned to receive jtpa training did not have significantly 
greater earnings than control group members 5 years after their 
assignment. For some of the four targeted worker categories— adult men, 
adult women, male youths, and female youths — treatment group earnings 
exceeded those of the control group in some of the intervening years, but 
any statistically significant effects disappeared by the fifth year.“ 



Earnings Outcomes for 
Adult Men 



Annual earnings of adult men increased in each year following assignment 
for both the treatment and control groups. As shown in figure 1, in the first 
year after assignment, the average annual earnings of adult men in the 
treatment group grew from about $4,400 to about $6,900. This group s 
earnings continued to rise in the subsequent years, reaching 
approximately $8,700 in the fifth year after assignment to receive jtpa 
training. The earnings of adult men in the control group, which did not 
receive JTPA training, also rose following assignment, but this group’s 
earnings were less than those of the treatment group for each of the 5 
years. 



"Soc Howarti S. Hloom aiul othiTS, Tlu- Nati ona l JTPA Study: Title 1 1-A Impa cLs o n Kiirningsand 
Empl oymont at 18 M onths (Bi’thi'sda, Mcl.: Abt Associates, Inc., Jan. 190.5). 

"See Uirry L. Orr and others. The National .ITPA Study: Impacts, Benefits, and Costs of Title II-A 
(liethesda, Md.: Alit Associates, Inc., Mar. 1094) for the results at JUinontlis. 

"’Bloom iuid others. Tlie NatiomU .ITPA Study: Title* II-A hopfcts on EaniinRs and Employwnt at 18 
Montlts and Orr and ofhersVTlie National JTPA St udy: Impacts, BeneTits^ iuul Cos ts of Title 11-A. 

‘ 'Generally, <>ur results matched the findinRs of the N.IS for the 150 months immediately following 
assignment. 

G 



o 
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Figure 1: Earningi of Adult Men Before 
and After Assignm^^nt 
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After 5 years, the difference between earnings of the treatment and control 
groups was not statistically significant. Five years after assignment, the 
treatment group’s earnings had exceeded those of the control group by 
approximately $300 to $500 annually, but only in the first 3 years were 
these differences statistically significant. 



earnings of adult women assigned to the treatment group increased from 
approximately $2,800 in the year of assignment to approximately $4,700 in 
the first year following assignment. This group’s earnings continued to 
climb, reaching ^proximately $6,600 in the fifth year. Earnings of adult 
women in the control group followed a similar pattern, but this group’s 
earnings were lower than those of the treatment group in each year, 
reaching approximately $6,200 during the fifth year. 







5 



Years Before and After Random Assignment 



Treatment Group 
Control Group 



Earnings Outcomes for 
Adult Women 



Earnings of adult women showed a pattern similar to those of adult men, 
increasing in each year after assignment. Figure 2 shows that the annual 
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Figure 2: Earnings of Adult Women 

Before and After Assignment 9ooo Annual Eamings 
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As with the earnings of adult men, 5 years after assignment the difference 
between the treatment and control groups’ annual eamings was not 
statistically significant. However, during the first 4 years after assignment, 
the differences between the treatment and the conU'ol groups’ eamings 
were statistically significant in each year, with treatment group eamings 
approximately $300 to $600 higher than control group eamings annually. 



Earnings Outconies for The earnings of male youths in the control group, like those of adult men 

Malo Youths adult women, increased in each year following assignment. Figure 3 

shows that the ear nings of male youths in the treatment group increased 
from approximately $2,900 in the year of assignment to approximately 
$4,600 in the first year after assignment. This group’s eai'nings continued 
to grow during the .5-year period, reaching a high of approximately $7,600 
in the fifth year. The earnings of male youths in the control group also rose 
during the .5-ycar peiiod following assignment, climbing from 
approximately $4,<S00 in the first year to approximately $6,800 in the fifth 
year. 
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Figure 3: Earnings of Male Youths 
Before and After Assignment 



Earnings Outcomes for 
Female Youths 
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Annual Earnings 




Years Before and After Random Assignment 



Treatrnent Group 
Control Group 



We found no significant difference between the treatment and control 
groups’ annual earnings 5 years after assignment. Although the control 
group’s earnings were higher than the treatment group’s during the first 3 
years following assignment, the differences, which ranged from 
approximately $200 to $400 each year, were not statistically significant. 
During the fourth and fifth years, the treatment group had higher earnings 
than the control group, but these differences toe were not statistically 
significant. 



Earnings of female youths showed a pattern similar to that of male youths, 
growing in each year following assignment. Earnings of female youths in 
the treatment group rose from approximately $2,000 duiing the year of 
assignment to approximately $3,300 in the first year following assignment 
(see fig. 4). This group’s earnings continued to climb, reaching 
approximately $5,400 in the fifth year following assignment. 'The earnings 
of female yout hs in the control group also rose duiing the 5-yem' peiiod. 
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climbing from approximately $3,400 in the first year to a high of 
approximately $5,200 in the fifth year. 



Figure 4: Earnings of Female Youths 
Before and After Assignment 
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We found no significant differences between the treatment and control 
groups’ annual earnings 5 years after receiving their assignments. During 
the first 2 years following assignment, the control group’s earnings were 
higher than the treatment group’s, but the differences of less than $100 
annually were not statistically significant. In the fourth and fifth years 
following assignment, the treatment group had earnings of approximately 
$100 to $300 higher than the control group, but these differences also were 
not statistically significant. 
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As with earnings, employment rates of those assigned to receive jtpa 
training were not significantly greater than employment rai es of control 
group members 5 years after assignment. For some of the four targeted 
worker categories, treatment group employment rates were higher than 
those of the control group in some years, but any statistically significant 
effects dis^peared by the fifth year. 



Employment Rates of The employment rates of both treatment and control group adult men 

Adult Men peaked during the calendar year of assignment and then declined in 

subsequent years, eventually reaching levels lower than those of the men 
before entering the study (see fig. 5). For example, the employment rate 
for adult men in the treatment group was 87 percent in the year of 
assignment. The percent employed declined in the following years, 
reaching 72 percent by the fifth year following assignment, which was 
low'er than the group’s employment rate of 79 percent in the year before 
entering the study. The adult men in the control group showed a similar 
pattern — their employment rate was 87 percent in the year of assignment 
but dropped to 71 percent in the fifth year after assignment. 



Employment Rates of 
Participants I^ot 
Significantly Greater 
Than Control Group 
Employment Rates 
After 5 Years 
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Figure 5: Employment Rates of Adult 
Men Before and After Assignment 
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After 5 years, the difference between the treatment and control groups’ 
employment rates was not statistically significant. The treatment group’s 
employment rates were higher than the control group’s in each year 
following assignment, although the differences in the employment rates 
were statistically significant only in the fourth year following assignment. 



The pattern of employment rates of adult women was somewhat simQar to 
that of adult men. The employment rates of adult women were highest 
during the calendar year following assignment, with 80 percent of the 
treatment group and 77 percent of the control group employed (see fig. 6). 
After the first year, however, the employment rates for both the treatment 
and control groups fell, reaching 69 percent and 67 percent, respectively, 
in the fifth year following assignment. These rates in the fifth year were 
also lower than each group’s employment rate in the year before 
assignment. 
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Figure 6: Employment Rates of Adult 

Women Before and After Assignment 95 Percent Employed 
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We found no significant differences between the treatment and control 
groups’ employment rates 5 years after assignment. The treatment group’s 
employment rates exceeded the control group’s in all 5 years following 
assignment, usually by about 2 to 3 percent, but only in the first 3 years 
were these differences statistic^y significant. 



EmployrriBnt Ratos of Malo The pattern of employment rates of male youths was somewhat similar to 

Youths that of adult men and women: the male youths’ employment rates peaked 

during the calendar year foUowing assignment — reaching nearly 
91 percent for the treatment group and over 92 percent for the control 
group — but then declined (see fig. 7). However, in contrast to the 
employment rates of adults, those of male youths were slightly higher 5 
years after assignment than before assignment, reaching 81 percent for the 
treatment group in the fifth year, compared with 80 percent in the year 
before assignment. 
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Figure 7: Employment Rates of Male 
Youth'i Before and After Assignment 
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Female Youths 
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We found no significant differences between the treatment and control 
groups’ employment rates 5 years after assignment. 'AThile the employment 
rates for the control group actually exceeded those for the treatment 
group in the year of assignment and the first and third years following 
assignment, none of the differences were statistically significant. 



The employment rates of female youths in both the treatment and control 
groups pealced during the calendar year of assignment, declined somewhat 
over the next 4 year's, and then slightly increased in the fifth year (see fig. 
8). As with those of male youths, the employment rates of female youths 
were slightly higher 5 years after assignment than before assignment. The 
emplo 3 mient rates of female youths were 74 percent for the treatment 
group and 73 percent for the control group in the fifth year following 
assignment, compared with 71 and 73 percent, respectively, in the year 
before rissignment. 



PHgcl2 GAO/HEHS-96-40 JTPA Outcomes 



B-256749 



Figure 8: Employment Rates of Female 

Youths Before and After Assignment 95 percent Employed 

90 
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We found no significant differences between the treatment and control 
groups’ employment rates 5 years after assignment. Employment rates for 
the treatment group exceeded those for the control group in 4 of the 5 
yeturs following assignment, but none of the differences in employment 
rates were statistically significant. 



Conclusions 



Though both long-term earnings and employment rates for njs treatment 
groups surpassed those for them respective control groups, the differences 
did not meet our test for statistical significance. Five years after 
expressing an interest in ,iTPA-sponsored job training, individuals assigned 
to participate in the program did not have earnings or employment rates 
significantly higher than individuals not assigned to participate. 



Agency Comments 



In commenting on a draft of this report. Labor expressed several concerns. 
It took exception to what it characterized as unwarranted negative 
conclu-sions that are not consistent with the report findings. Labor also 
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took issue with the importance the report places on tests of statistical 
significance applied to earnings of an individual group in a given year, 
preferring to emphasize other evidence of the positive effect of jtpa on 
participant earnings over the 5-year period. Labor also expressed concerns 
that the report findings have limited relevance to current job traimng 
programs. 

We believe that our conclusions are well supported by our findings. C 
several occasions where appropriate, we have noted comparisons 
favorable to the jtpa treatment groups, including in the “Results in Brier 
and “Conclusions” sections. Although other evidence covering the 5-year 
period might be found to better highlight the positive effects of jtpa 
training, our research focused on the earnings and employment rates of 
each target group in the fifth year after applying for jtpa training. Also, we 
do not believe that current or proposed job training programs sufficiently 
differ from jtpa training at the time of the njs to limit the relevance of our 
report findings. 



In its response. Labor enclosed an attachment with specific comments on 
the report and additional information. This attachment and our evaluation 
of the comments appear in appendix III. Labor also provided us with 
technical comments, which we have incorporated in the report where 
appropriate. 



As Eigreed with your office, unless you publicly announce its contents 
earlier, we plan no further distribution of this report until 30 days from the 
date of this letter. At that time, we will send copies to interested parties 
and make copies available to others upon request 



ir, 
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This report was prepared under the direction of Wayne B. Upshaw, 
Assistant Director, who may be reached on (202) 612-7006 if you or your 
staff have any questions. Gene Kuehneman, Senior Economist, 

(202) 512-4091, Jill Schamberger, Senior Evaluator, and Thomas L. 
Hungerford, Serrior Economist, were major contributors to this report. 

Sincerely yours, 




Cornelia M. Blanchette 
Associate Director, Education 



and Employment Issues 
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Appendix I 



Data Sources and Methodology 



To select our sample of individuals assigned to receive training under jtpa 
and our control group, we used participation informeition from the 
National jtpa Study (njs). We then obtained long-teim earnings and 
employment informeition for these individuals from ssa. Our analysis 
compared earnings and employment levels of individuals in the treatment 
and control groups to determine whether differences between these 
groups were statistically discemable. 



National JTPA Study 



The original njs data set contained demogr^hic and program information 
on 20,601 people who applied for jtpa services between November 1987 
and September 1989 in 16 local service delivery ai'eas. Program ^plicants 
were recruited, screened to determine their eligibility, assessed to 
determine their service needs smd wants, and recommended for services. 
NJS participants were then randomly assigned to either the treatment 
group, which was allowed to participate in jtpa title II-A programs, or the 
control group, which was not allowed to participate in these programs for 
18 months. Approximately two-thirds of the ^plicants were assigned to 
the treatment group and one-third to the control group. The control and 
treatment groups were closely matched in demogr^hic variables such as 
age, race, and education, which typically allows a meaningful comparison 
of average outcomes between the two groups. 

However, two factors intervened to make such a compari.son problematic. 
First, not aU members of the treatment group participated in jtpa 
programs. For example, about two-thirds of the adult treatment group 
members enrolled in jtpa, but the other one-third either found jobs on 
their own or decided not to participate in the program. Second, a 
substantial minority of the control group members chose to participate in 
some alternative, non-jTPA training programs. These complications 
preclude attributing earnings differences between the two groups solely to 
JTPA training. Therefore, our findings refer to differences between the 
treatment and control groups rather than between individuals who did or 
did not receive jtpa training. Furthermore, we do not know which of the 
control and treatment group members chose to receive training later than 
18 months after assignment. 

The NJS was not designed to track treatment or control group members 
beyond 30 mont hs. Therefore, to calculate and compare longer teim 
earnings and employment outcomes for these groups, we needed 
information from another source. 
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Data Sources and Methodology 



Social Security 
Earnings Records 



We obtained annual earnings records from ssa for the individuals in the njs 
treatment and control groups, ssa maintains information on annual 
eanungs of individuals contributing to either Social Security or Medicare. 
We assumed that an individual was employed if his or her ssa records 
showed positive earnings for a given year. We adjusted data for what we 
assumed were data entry or processing errors, and we also rounded 
reported negative earnings to zero. 



Data Analysis 



We analyzed the njs and ssa earnings records of 13,699 njs participants*^ to 
determine their annual earnings and employment outcomes for the 3 years 
before assignment to the treatment or control group, the year of 
assignment, and 5 years following assignment. The 3 years of prior 
earnings and employment data served to demonstrate the prior 
comparability of treatment and control groups. The 5 years of 
postassignment data effectively doubled the 30-month follow-up period for 
the NJS. The treatment group had 9,275 individuals, and the control group 
had 4,424 individuals. 

We used individual earnings data and calculated means and variances for 
each of the four target groups — adult men, adult women, male youths, and 
female youths — to compare the treatment groups’ earnings and 
employment outcomes with those of the control groups. We tested for 
differences in earnings and employment outcomes at the 5-percent 
significance level. 

We calculated annual earnings amounts using ssa information on Social 
Security-covered earnings for nonfederal workers and on earnings covered 
by Medicare for federal workers. We calculated employment rates as the 
percentage of each group with positive covered earnings in a calendar 
year. Individuals with unreported earnings may have had their earnings 
and employment understated in our analysis. Individuals whose earnings 
exceeded the Social Security withholding ceiling may also have had their 
earnings understated in our analysis.*^ These limitations applied to both 
the treatment, and control groups, and we do not believe they affected the 
two groups differently. 



did not analyze <*arninRs and employment information for the NJS participants who were 
as.sij»n(‘d aft(*r hiKS brcjius<* w<* had <)iily 4 y(*ars of t)oslas,sijtnment data for them. We also excluded 
tlie NJS piulicipant.s whose SSA records did not adequately match infonnation collected for the NJS, 
such as nmiK* or birth month ;uul yeiu*. 

' ‘( )nly a certain amount of workers* eantiitjts — for example $55,500 in 1992— were subject to the Social 
S('curity tax. 
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Data and Statistics for Figures 



Table II.1: Data Points for Figure 
1 — Earnings of Adult Men Before and 
After Assignment 



Table 11.2: Data Points for Figure 
2— Earnings of Adult Women Before 
and After Assignment 





Annual earnings 




Statistically 
significant 
difference at 5% 


Time period 


Treatment group 


Control group 


level? 


3 years before 


$5,883 




$5,924 


no 


2 years before 


5,680 




5,894 


no 


1 year before 


5,106 




5,246 


no 


Assignment 


4,439 




4,242 


no 


1 year after 


6,901 




6,410 


yes 


2 years after 


7,792 




7,254 


yes 


3 years after 


7,936 




7,363 


yes 


4 years after 


8,282 




7,725 


no 


5 years after 


8,651 




8,326 


no 








Annual earnings 




Statistically 
significant 
difference at 5% 


Time period 


Treatment group 


Control group 


level? 


3 years before 


$3,262 




$3,020 


no 


2 years before 


3,377 




3,215 


no 


1 year before 


3,230 




3,048 


no 


Assignment 


2,823 




2,703 


no 


1 year after 


4,702 




4,323 


yes 


2 years after 


5,705 




5,047 


yes 


3 years after 


5,902 




5,319 


yes 


4 years after 


6,367 




5,811 


yes 


5 years after 


6,556 




6,154 


no 
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Table 11.3: Data Points for Figure 
3— Earnings of Male Youths Before 
and After Assignment 



Table 11.4: Data Points for Figure 
4 — Earnings of Female Youths Before 
and After Assignment 





Annual earnings 




Statistically 
significant 
difference at 5% 


Time period 


Treatment group 


Control group 


level? 


3 years befo.e 


$860 




$828 


no 


2 years before 


1,456 




1,575 


no 


1 year before 


2,179 




2,303 


no 


Assignment 


2,894 




3,014 


• no 


1 year after 


4,612 




4,792 


no 


2 years after 


5,620 




5,963 


no 


3 years after 


6,130 




6,497 


no 


4 years after 


6,687 




6,425 


no 


5 years a^ter 


7,554 




6,778 


no 








Annual earnings 




Statistically 
significant 
difference at 5% 


Time period 


Treatment group 


Control group 


level? 


3 years before 


$629 




$663 


no 


2 years before 


1,069 




1,090 


no 


1 year before 


1,529 




1 ,707 


no 


Assignment 


1,974 




2,098 


no 


1 y ar after 


3,339 




3,389 


no 


2 years after 


4,045 




4,125 


no 


3 years after 


4,393 




4,383 


no 


4 years after 


4,934 




4,610 


no 


5 years after 


5,433 




5,209 


no 
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Table II.5: Data Points for Figure 
5 — Employment Rates of Adult Men 
Before and After Assignment 



Table II.6: Data Points for Figure 
6 — Employment Rates of Adult Women 
Before and After Assignment 



O 

ERIC 





Employment rates 
(percent employed) 




Statistically 
significant 
difference at 5% 


Time period 


Treatment group Control group 


level? 


3 years before 


79.2 


78.8 


no 


2 years before 


77.6 


79.0 


no 


1 year before 


78.3 


80.6 


no 


Assignment 


87.0 


87.4 


no 


1 year after 


86.7 


86.4 


no 


2 years after 


81.3 


80.7 


no 


3 years after 


76.3 


73.9 


no 


4 years after 


73.7 


70.0 


yes 


5 years after 


72.3 


71.1 


no 








Employment rate 
(percent employed) 




Statistically 
significant 
difference at 5% 


Time period 


Treatment group Control group 


level? 


3 years before 


65.8 


65.8 


no 


2 years before 


67.1 


66.2 


no 


1 year before 


69.4 


68.4 


no 


Assignment 


76.7 


75.7 


no 


1 year after 


80.3 


7.4 


yes 


2 years after 


76.3 


73.7 


yes 


3 years a'tor 


71.2 


68.1 


yes 


4 years after 


69.7 


67.7 


no 


5 years after 


68.7 


67.4 


no 
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Table H.7: Data Points for Figure 
7 — Employment Rates of Male Youths 
Before and After Assignment 



Table 11.8: Data Points for Figure 
8 — Employment Rates of Female 
Youths Before and After Assignment 



Time period 


Employment rate 
(percent employed) 

Treatment group Control group 


Statistically 
significant 
difference at 5% 
level? 


3 years before 


46,5 


48.3 


no 


2 years before 


. 63.2 


66.4 


no 


1 year before 


79.6 


79.6 


no 


Assignment 


89,2 


91.8 


no 


1 year after 


eo.5 


92.1 


no 


2 years after 


88,4 


87.8 


no 


3 years after 


82,2 


82.6 


no 


4 years after 


80,4 


79.4 


no 


5 years after 


81,1 


77,5 


no 








Employment rate 




Statistically 

significant 




(percent employed) 




difference at 5% 


Time period 


Treatment group Control group 


level? 


3 years before 


41,2 


43.6 


no 


2 years before 


57.6 


60,5 


no 


1 year before 


70,7 


72.8 


no 


Assignment 


82.0 


81.8 


no 


1 year after 


82.0 


79.6 


no 


2 years after 


79.7 


78.2 


no 


3 years after 


73.8 


75.1 


no 


4 years after 


71,7 


70.7 


no 


5 years after 


73.9 


73.0 


no 
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Table 11.9: Standard Errors Associated 
With Earnings of Adult Men Before and 
After Assignment 



Table 11.10: Standard Errors 
Associated With Earnings of Adult 
Women Before and After Assignment 



O 





Standard error 



Time period 


Treatment 


Control 


Pooled error 


T-ratio 


3 years before 


6,740 


6,983 


221 


-0.19 


2 years before 


6,547 


6,702 


213 


-1.00 


1 year before 


5,818 


5,779 


188 


-0.74 


Assignment 


4,724 


4,405 


149 


1.32 


1 year after 


6,825 


6,406 


216 


2.27 


2 years after 


8.029 


7,516 


254 


2.12 


3 years after 


8,753 


8,335 


279 


2.06 


4 years after 


9,446 


9,117 


302 


1.84 


5 years after 


10,134 


9,611 


322 


1.01 


Subgroup size 


2.874 


1,435 












Standard error 






Time period 


Treatment 


Control 


Pooled error 


T-ratio 


3 years before 


4,638 


4,412 


134 


1.81 


2 years before 


4,714 


4,664 


137 


1.18 


1 year before 


4,315 


4,296 


126 


1.44 


Assignment 


3,297 


3,128 


95 


1.27 


1 year after 


5,017 


4,828 


145 


2.62 


2 years after 


6,242 


5,707 


178 


3.70 


3 years after 


6,829 


6,414 


196 


2.98 


4 years after 


7.646 


7,039 


218 


2.55 


5 years after 


7,797 


7,480 


225 


1.79 


Subgroup size 


3,631 


1,723 
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Table 11.11: Standard Errors 
Associated With Earnings of Male 
Youths Before and After Assignment 












Time period 


Standard error 
Treatment Control 


Pooled error 


T-ratio 




3 years before 


1,689 


1,687 


87 


0.36 




2 years before 


2,339 


2,604 


125 


-0.96 




1 year before 


2,762 


2,778 


142 


-0.88 




Assignment 


3,048 


2,902 


154 


-0.78 




1 year after 


4,799 


4,603 


243 


-0.74 




2 years after 


5,807 


5,703 


297 


-1.16 




3 years after 


6,561 


6,646 


338 


-1.08 




4 years after 


7,445 


7,186 


378 


0.69 




5 years after 


8,150 


7,807 


413 


1.88 




Subgroup size 


1,177 


559 







Table IL12: Standard Errors 
Associated With Earnings of Female 
Youths Before and After Assignment 





Standard error 






Time period 


Treatment 


Control 


Pooled error 


T-ratio 


3 years before 


1,327 


1,392 


61 


-0.56 


2 years before 


1,885 


1,797 


84 


-0.25 


1 year before 


2,113 


2,352 


99 


-1.80 


Assignment 


2,194 


2,307 


101 


-1.23 


1 year after 


3,828 


3,767 


172 


-0.29 


2 years after 


4,612 


4,663 


209 


-0.38 


3 years after 


5,271 


5,388 


240 


0.04 


4 years after 


6,053 


5,803 


270 


1.20 


5 years after 


6,494 


6,309 


291 


0.77 


Subgroup size 


1,593" 


707 







"Subgroup size for the treatment group 3 years after assignment is 1 ,592. 
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Note: GAO comments 
supplementing those in the 
report text appear at the 
end of this appendix. 



See comment 17. 




U.S. Department of Labor Assistant Secretary for 

Emp oymcnt and Training 
Washington. 0 C 20210 



.NOV 30 I996 

Ms. Cornelia Blanchette 
Associate Director 
Education and Employment Issues 
U.S. General Accounting Office 
Washington, D.C. 20548 

Dear Ms. Blanchette: 

The Department of Labor appreciates the opportunity to comment on the General 
Accounting Office (GAO) report on the long-term earnings and employmem outcomes of Job 
Training Partnership Act (JTPA) Title 0 paitictpiants. We think that the study is well- 
designed; however, we have a number of concerns with this report. Our main issues are 
summarized below. We have attached more detailed technical comments from our staff and 
academic reviewers. 

We believe that the report’s conclusions present an unwarranted negative picture of the 
effectiveness of JTPA adult programs that are not consistent with the findings presented in 
the report. They also fail to recognize the limited relevarvcc of those findings to current job 
training programs. At the time the study was conducted - participants examined by GAO 
enrolled in the program during 1987-1989 — the average federal investment in JTPA partici- 
pants was only $2,400 per participant. The impacts in your rqx)rt imply that the bmefits 
outweigh the cost and ^so suggest a return from JTPA comparable to the return of a year of 
postsocondary education. Moreover, these returns were achieved with training lasting only 
about three and a half months on average and with a group that is more disadvantaged than 
the typical college student and therefore, would be expect^ to require a larger investment 
to achieve, substantial returns. 

The GAO report finds consistently positive impacts of JTPA on earnings of adult men 
and women throughout the five year period. For both male and female youths, the pattern 
of negative impacts early in the period is followed by consistently positive impacts. Such 
patterns ere un.Ukdy to have occurred by chance. Thus, despite the results of individual 
statistical significancv't tests which are emphasized, there is good evidence that JTPA 
increased the earnings of its participants. 

In considerihg the implications of these fuidings it is important to recognize that 
there have been many changes in JTPA since 1989. We have learned a great deal about ' it 
makes a job training program sujxessful from careftiUy conducted and evaluated demonstra- 
tion programs such as JOBSTART and the Minority Female Single Parent Demonstration. 
Implementation of lessons learned from such efforts and major amendments to the legislation 




Page 26 



26 



GAO/HEHS-96-40 JTPA Outcomes 



Appendix III 

Comments From the Department of Labor 
and Our Evaluation 



- 2 - 



have resulted in the current JTPA program serving a more at-risk population with more 
intensive services than was the case in the period covered by the GAO report. We are 
confident that a careful evaluation of JTTPA, if conducted on the current program, would 
show laiger impacts than those found in this study. 

Finally, there are several technical issues that are amplified in the attachment, but which 
point to an overemphasis on statistical significance tests without disclosure of the magnitude 
of the standard errors. Statistical significance is not clear-cut, it r^resents a continuum of 
acceptability. Basic standarcls of research dictate that the report should include in each table 
the sample sizes for each tirget group, the size of the treatment and control groups for these 
target groups, the standard enon, and significance levels. 

Again, we thank you for the opportunity to comment, and we appreciate your taking 
these comments into consideration as you finalize the report. 



Sincerely, 




Timothy M. Bamiclc 
Assistant Secretary of Labor 



Attachment 
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See comment 1 . 
See comment 1 



See comment 2. 



See comment 2. 



See comment 3. 



See comment 4. 



DOL Comments on GAO Report 

The General Accounting Office (GAO) in this report uses social security earnings data to 
examine the long-term net impact of JTPA training provided to adults and out-of-school youth. 
This GAO Study extends to five years the 30-month follow-up information on JTPA impacts 
information that DOL published last year. DOL'a comments on this GAO R^it aie as follows: 

1. The report undenutes the gross returns to JTPA training of those that received 
training. 

o The gross returns to JTPA participation for adults exceed or are 
comparable to the gross returns from postsecondary education reported in 
the literature. In particular, for women they exceed Kane and Rouse*s 
estimated gross return of 5 % for a year of credit at either a community 
college or a four-year collide. Further, these gross returns were achieved 
with a group that is more disadvantaged, in less time (three to four months 
of training), and at a lower cost (Abt reported the net incremental cost 
was $1146 for adult women and $1509 for adult men) than a year of 
postsecondary education. 

2. There are favorable aspects to the findings in the study that are not acknowledged in 
the report. 

o In all four target groups-adult men, adult women, male youth, and female 
youth-participant earnings are higher in the fifth year than the control 
group. For both adult men and adult women, the participant earnings are 
higher than controls throughout the five-year post-program period. For 
male youth, there is a positive trend and the fifth-year earnings of 
participants are over 10 pcrcetit above those of controls. 

o If the net impacts ate accumulated over lime during the five-year follow- 
up— which GAO does not do in this report-thc net benefits of the program 
clearly outweigh the costs of the program for both adult men and adult 
women. 



3. A major conclusion of the is that JTPA impacts on adults aie dissipating over 
time. GAO’s estimates do not support this conclusion. 

o The increase in standard errors s^pean to be equally if not more 
re^nsible for tlie decline in statistical significance than the change in the 
estimated impacts. For example, adult males in the treatment group 4 
yean after training had $557 higher earnings than controls, which was not 
statistically significant at the .05 level. But the treatment effect for adult 
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See comment 5. 
See comment 5. 

See comment 6. 

See comment 7 

See comment 8. 
See comment 9. 
See comment 10. 
See comment 7 

See comment 1 1 



See comment 1 1 . 



2 

males in the first year after training was a smaller $491, which was 
statistically significant at the .05 level. 

o Obviously, the imprecision of the estimates is rising over time. To 
conclude that the training effects dissipate over time, the study must test 
whether the training effect is different in year 5 than it was in year 1. 

4. The rqx>it emphasizes the individual year-by-year tests of statistical significance; 
however, its conclusions calling Into question the usefulness of the program in improving 
participants long-term earnings and employment prospects are much broader. 

o These conclusions require the analysis of the total impact on earnings and an 
assessment of the overall costs and benefits, not a comparison of earnings levels 
at a specific point in time. These analyses were not provided in the report, and 
based upon the information provided in the tables, the conclusions for ^ults are 
misleading. While the positive fifth year earnings impacts are statistically 
insignificam for' each of the four target groups, the odds of all four being positive 
purely by chance are 6.25 percent. 

5 . The report docs not mention the magnitude of the standard errors. The tables should 
list 95 percent confidence intervals for each estimated training effect. 

o The report also treats figures that are not significant at the 5 percent level 
as if they are equal to zero. If the significance levels are between the 5% 
and 10 %, it is misleading to treat the values as zero. Statistical 
significance is not a knife edge of yes or no, but a continuum. The r^rt 
should discuss the results, the probability values, and changes in the 
significance levels. At a minimum the appendix tables should include the 
sample size (for men, women, male youths and female youths), the size 
of the treatment and control groups for these target groups, the standard 
errors, and specify significance levels (1%, 5 %, 105(). 

6. The report notes that a third of the group assigned to the treatment group never 
enrolled in JTPA and that a substantial minority of the control group received some form 
of non-JTPA training. The conclusion that "these complications preclude solely 
attributing earnings differences between the two groups to JTPA training" is incorrect. 

o The fact that a third of the treatment group never enrolled in JTPA and 
that a substantial miiKirity of control group members received some form 
of non-JTPA training and the additional fact that control group memben 
were allowed to enroll in JTPA after 18 months ail imply that the impacts 
estimated by GAO understate the total Impact of JTPA on participants' 
eanungs and empbyment. 
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o A JTPA effect which adjusts for this understateracnt can be estimated by 
making the assumption that non-paitkipating treatment gxx)up members 
received no benefit from JTPA. In the 30-month report, DOL railed 
both pure assignee impacts and corrected enroUcc impacts. Correcting for 
non-paiticipating treatment group members boosts treatment effects by 
roughly one-third--making the impact close to $1,000 in some years for 
aduh men and women and for young males in the filth year of follow-up. 

7. This study reports results only for the four major target groups. GAO will be 
providing access to the social security data to DOL's contractor for additional analysis, 
including examining the results for sub-groups. 

o Given the 30-month study results and some preliminary work done using 
XJI wage records data, most likely some sub-groups will show quite 
favorable 5-year net impacts. For example, adult women who were 
receiving AFDC at entry and assigned to OJT showed markedly positive 
impacts in the 30-month follow-up, and we expect such impacts will 
continue in the S-year results. 

o Also, our preliminary work with UI wage records data suggest that youth 
with prior arrest records may show strong positive impacts in the fourth 
and fifth year# of foUow-up. Ihis fmding for youth with prior arrest 
records would be consistent with the positive results found for such youth 
in the JOBSTART denmnstration. 

8. While it is disappointing that the impacts throughout the follow-up period are not 
larger for each of tte four target groups, these results serve to highlight the disparity 
between the amount spent in these programs during the observation period and our 
current expectations of what these programs can accomplish. 

o The investment put into participants by JTPA during 1987 to 1989 when 
this sample was enrolled was very modest both in tenns of dollars and 
time. An average federal investment of $2,400 was spent on participants 
during this period in JTPA, and the median length of training was three 
and a half months. 

o Taking into account that some controls found training on their own, the 
incremental cost of the program was $ 1 ,200 for adult women and $850 for 
adult males. The earnings gains reported in this study of $500 a year for 
five yean yield a positive return for this small investment. 

o Achieving impacts sufficient to boost persons out of poverty-say, $5 ,000 
a year throughout a person's working life-is totally unrealistic for a one- 
time investment of $1,000 or $2,000. If we want to have a reasonable 
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chance of moving people out of poverty, then we need to invest more in 
paiticipants. 

o States and localities need to keep this in mind as new legislation gives 
them much more control of the design and operation of their job training 
programs. The current JTPA serves a more at*risk population with more 
intensive services than the JTPA in the 1987>89 period covered by this 
report. 

o DOL expects that the current JTPA would produce larger impacts than the 
earlier JTPA. States and localities will need to make their programs at 
least as intensive as the current JTPA if they expect to achieve greater 
impacts than those found here. 

9. The findings in this GAO report should be examined in combination with results from 
other random assignment studies of more intensive employment and training programs. 
There is a fair number of programs that now have been evaluated using random 
usignment. 

o While the results of early State welfaie-lo-work programs were somewhat 
mixed, the GAIN evaluation is showing strong impacts through the third 
follow-up year. The average third-year net impact across all six sites in 
the GAIN evaluation is an increase in earnings of $628-represcnting an 
increase in camings of 25 percent over controls. In the Riverside site, 
participants earned over a $ 1 ,000 more than controls during the third year 
of foUow-up-an increase in earnings of 40 percent over controls. 

o The JOBSTART demonstration tested a fairly intensive Don-residential 
program for out-of-school youth. The results were disappointing for the 
sample as a whole, but one site achieved a net earnings gain for 
participants over controls of over $3,000 a year during the third and 
fourth years of follow-up-r^rcsenting a 50 percent increase in earnings 
of paiticipants over controls. Further, among males with arrest records 
prior to entering the JOBSTART program at all sites, participants earned 
$1 ,200 more than controls during the fourth year of foUow-up*-an increase 
in earnings of 28 percent over controls. 

o The Minority Female Single Parent Demonstration provided job training 
to women in their 20s and 30s in four sites. The site that did so well in 
the JOBSTART demonstration for youth-the Center for Employment and 
Training (CBT) in San Jose-also was the best performing program in this 
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See comment 16. 
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demonstration aimed at young adults. Earnings impacts of over a $l ,000 
a year above controls were maintained through the fifth year of follow-up- 
- representing a 16 percent increase in earnings over controls. The GET 
program is distinguished from other job training programs in the extent 
of its links to the private sector both in terms of instruction and 
placement. 

10. GAO*s reported findings are for a subsampic of 13,799, which is only 67 percent of 
the 20,601 total sample. About a third of the sample was excluded because there were 
only four years of follow-up datt for the persons who were randomly assigned after 
1988. 

o It is possible that employment and earnings differ across individuals 
assigned in earlier and later years due to changing economic conditions 
(e.g. the most recent recession). GAO should provide information on the 
full sample for the first four years, in addition to the information provided 
for five years on two-thiids of the sample. 

1 1 . GAO should explain that the measure of employment rates contained in this report 
is not the definition that is generally reported in government statistics. The definition in 
the report is the proportion of the sample ever employed during a year as this is the only 
employment rate that may be calculated with the Social Security data. However, GAO 
should iKJtc that the reported levels are not comparable to figures reported in other 
sources such as the CPS and BL^ reports. 

12. DOL*s experience using social security data in our old Continuous Manpower 
Longitudinal Survey suggests that the most recent year of available social security 
records is subject to considerable corrections the following year. This somewhat calls 
into doubt both the fifth year declines in impacts for adult men and women and the fifth 
year iiKiease in impact for male youth. 
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The following are gao’s comments on the Department of Labor’s letter 
dated November 30, 1995. 



GAO Comments 



1. Labor comments that our report understates the gross returns to jtpa 
training. Furthermore, Labor implies that these gross returns calculations 
compare favorably with the returns to college education. 

Our objective, as clearly stated in the report, was not to evaluate the 
cost-effectiveness of jtpa training, but rather to determine and compare 
the long-term effects of .jtpa training. In fact, because we did not calculate 
the gross return to jtpa participants the report cannot have understated or 
overstated the values. Such calculations were not within the scope of this 
report. While it may be true that these retmms are favorable, we have no 
basis to judge the favorability of the gross returns to training. 

2. Labor states that the report does not acknowledge favorable aspects of 
this study. Specifically, Labor cited that (1) aU four target groups had 
higher earnings in the fifth year after assignment; (2) both adult treatment 
groups had higher earnings than their respective control groups in each of 
the 5 years following assignment; and (3) for male youths, a positive trend 
exists, and the fifth-year earnings exceed those of the control group by 
over 10 percent. 

Contrary to Labor’s comment, we did note many of these favorable 
program outcomes in om' report. We stated that adult male treatment 
group members had higher earnings than adult male control group 
members and presented similar findings for the other three target groups. 
We further stated that adult male treatment group earnings exceeded 
control group earnings in each of the 5 years and reported similar 
information for adult women. Also, we noted the positive trend for 
earnings of male youths. We did not note the percentage difference for 
male youths in the fifth year becau.se we did not report percent^e 
comparisons for any of the target groups. 

3. Labor states that if the training impacts are accumulated over time 
during the 5-year follow-up period, the net benefits outweigh the costs. 

As we stated in comment 1 , our objective was not to evaluate the 
cost-effectiveness of .rrPA training, but rather to determine and compare 
the long-term effects of .hta training. Wliile it may be true that the net 
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benefits outweigh the costs, we have no basis to judge this because such 
calculations were beyond the scope of this report. 

4. Labor states that the increase in standard errors is primaiily responsible 
for the decline in statistical significance of the estimat,ed impacts. 

While Labor is correct in stating that the standard errors were great er in 
the fifth year, it is not accurate to attribute a decline in statistical 
significance to either the estimated training effects or to the standard 
errors. The test statistics used for our significance tests are determined by 
the ratio of the estimates to their standard error, and attributing the lack of 
significance solely to either component of these ratios is inappropriate. 

5. Labor states that our conclusion requires assessing the total impact of 
.ITPA and the overall cost and benefits. TTie Department states that we 
overemphasize the importance of year-by-year significance tests in 
questioning the program’s usefulness in improving pailicipants’ long-term 
earnings prospects by stressing the insignificant effect of .itp.\ in the fifth 
year. 

We agree with Labor that year-by-year significance tests have limited value 
in assessing the total impact of jtpa and the overall cost and benefits. 
Furthermore, our year-by-year significance tests pro\ide statistical 
evidence that adult treatment group members achieved higher earnings for 
several years following assignment to jtpa training. While other evidence 
covering the 5-year period might be found to better highlight the positive 
effects of JTPA training, we chose to address the question of whether the 
fifth-year earnings of those assigned to participate in .hta differed 
significantly from the fifth-year earnings of those not assigned to 
participate in jtpa. 

6. While acknowledging that the observed earnings differences between 
the four target groups were not statistically significcUit in tiu* fifth year. 
Labor asserts that the odds that all four differences would be positive 
purely by chance is 6.25 percent.*'* 

This implies that an accumulation of not statistically significiuit 
observations provides more compelling empirical evidence tlian the actual 



'‘KoroxampU', if truly had nooffcct on oiiniings, tlu*ti tlu* pruhahiln.v iliai \ Uv rsiiinatcd 

difforonco would ho positive for any one target group is 50 perctMil - tluit is. half of th(‘ tmu* the 
estimate would he iiositive and the other hjilf it would Ix^ negative. Thus, tlu* ehaner that all four 
groups would show positive estimated earnings effects, if no true effe<-t existed foranv group, would 
l)e 50 percent t(j the fourth power or 0.25 percent. 
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significance test for any one group. While the probability (not odds) that 
all four not significajit fifth-year earnings differences would be positive 
purely by chance might be low, our research question is whether a 
significant earnings difference occurred for each target group. 

7. Labor comments that the report does not report the standard errors. 
Labor states that the report should include confidence intervals for the 
estimates, sample sizes, and standard errors and specify significance levels 
for the estimates. 

We have made several additions to tables in appendix II in response to this 
comment. We have added the sample size, the size of the treatment and 
control groups, the standard errors, and a reminder that the significance 
level chosen is 5 percent for the tables in this appendix. Since we have not 
presented point estimates of the earnings effects, we did not calculate 
confidence intervals for these estimates. Technical readers of our report 
can construct such estimates and the associated intervals from the 
information in the appendix II tables. 

8. Labor claims that the report treats figures that are not significant as 
zero. 

We do not report any training effect as zero. The magnitude of the earnings 
differences, whether significant or not, is discussed in the report and is 
easily calculated from the tables in appendix II. 

9. Labor states that statistical significance is not a knife edge of yes or no 
but a continuum. 

The level used for tests of statistical significance may be chosen from a 
broad range (or continuum) of values. Although different researchers may 
choose to use different values for the significance level, choosing a 
significance level before analyzing any data is common practice. Once this 
level has been chosen, statistical hypothesis testing very much involves a 
yes or no decision. Either the data reject the null hypothesis of no training 
effect at the set significance level or not. We follow these commonly 
accepted procedures for hypothesis testing, and our convention is to set 
the significance level for such tests at 5 percent. 

10. Labor also states that the report should discuss the results, the 
probability values, and changes in the significance levels. 
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Our report does discuss the results as weU as whether the earnings and 
employment effects were significant and whether this sigmficance 
changed over time. Although we do not present the probability values, 
technical readers of oiur report can calculate them using the information in 
the appendix II tables. 

11. Labor takes issue with our statement that complications (not all 
treatment group members received training and some control group 
members did receive training) precluded our attributing earmngs 
differences solely to jtpa training. It claims that these complications led us 
to understate the effect of ti-aining, implying that the earnings differences 
observed, along with perhaps some further overlooked earnings effects, 
can be attributed to jtpa. 

We clearly state that these complications preclude solely attributing the 
earnings effects to .rrPA training. However, we have no evidence that these 
factors led to an understatement of the effect of jtpa training. In the first 
place, a short delay can occur before an assignee can begin a training 
progranj. In some of these c ases, individuals find and accept employment 
instead of reporting for training. To the extent that these individuals are 
more fuUy employed and may earn more than they might have if they had 
attended jtpa training, our estimate may actually overstate the effect of 
training. Second and more importantly, if those who attend training are in 
some way more motivated than those who do not attend, it would be 
difficult to separate any increase in earnings due to training from the 
increase in earnings due to this motivation. At a minimum, we would need 
to identify which control group members were motivated to attend 
training to draw such inferences. 

12. Labor recommends adjusting the comparison by effectively renvving 
treatment group members who did not enroll in training. 

We chose to compare only those assigned to jti^a training with those not 
assigned to training to take full advantage of the original random 
assignment design. As we stated in comment 11, we would have needed to 
identify which control group members were motivated to attend training 
to justify removing the treatment group members who did not attend 
training. Since we could not take all the necessary steps to fully implement 
Labor’s suggestion, we chose not to make that or any other acijustments. 
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13. Labor comments that we will be providing the Department’s contractor 
with access to Social Security data for additional analysis, including 
exanuning the results for subgroups. 

We would like to clarify the details of this arrangement in light of the 
sensitive nature and confidentiality of individual eanungs records. When 
we began our work, Labor was also planning to evaluate the long-term 
impact of jtpa training on earnings. Both our and Labor’s evaluation 
(contracted out to Westat, Inc.) planned to use Social Security earnings 
records to supplement the information collected through the njs. In the 
spirit of cooperation. Labor requested and we agreed to provide 
aggregated eanungs data to the Department, which will submit computer 
programs to us; we will in turn run the programs and provide the output to 
Labor. Only aggregated information, such as means and standards 
deviations, will be reported. No data will be released that could be traced 
to individuals, nor will we provide Labor or its contractor with individual 
earnings records. 

14. Labor suggests that we include earnings and employment information 
for the third of the sample for whom only 4 years of follow-up data were 
available. 

We did not include this group in our analysis since we could not report on 
their earnings or employment 5 years after training. As such, any 
additional information provided would not address the question of 
whether jtpa had a long-term effect on the earnings or employment 
outcomes of the treatment group. 

15. Labor states that we should explain that our employment measure is 
not the definition that is generally reported in government statistics and is 
not compcirable to figures reported in Current Population Survey (cps) and 
Bureau of Labor Statistics (bls) reports. 

Our employment rate differs from measures reported in cps and bls 
reports but is appropriate for our purposes. Our employment rate is the 
number employed divided by all who were in the treatment or control 
group, which includes those workers who may have dropped out of the 
labor force. Since these workers presumably ^plied for training because 



' Tl\p Cl’S pounts ppople at any given time ius(l) eniploywl, (2) uneinployed, nr (J) not in the labor 
force — that is, not employed and not swking employment. The un'*mployment rate in the CPS is the 
number of unemployeri dividofi i>y the sum of the employed plus unemployed. If an employment rate 
were reported, it would bv one minus the unemployment rate, or, equivalently, the number employed 
divided by the sum of the employed plus iinemploye<l 
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they intended to keep working, we believe that all workers should be 
included in the denominator of the measure. Our measure also counts as 
employed everyone who worked during the year, even if they might have 
been unemployed for some portion of the year. As such, our measure may 
overstate the instantaneous employment outcomes of both the treatment 
and control groups relative to figures reported in CPS i_nd bls reports. 

16. Labor states that Social Security data are subject to considerable 
revisions in the first year of availability. It believes this calls into doubt 
fifth-year estimates for adult men, adult women, and male youths. 

While data are often subject to revision, we have no reason to suspect that 
the data for those assigned to training in 1988 are materially less reliable 
than for those assigned to training in 1987. The fifth year (1993) of 
earnings data for those assigned to training in 1988 was extracted from ssa 
records in March 1995. An ssa official responsible for updates and 
revisions to ssa earnings data said that we could expect the accuracy and 
completeness of our extract to exceed 99 percent. 

17. Liibor states that we fail to recognize the limit ed relevance of our 
findings to current job programs. 

While we agree that our evaluation has limitations, we disagree that it has 
little relevance to current job programs. We make it quite clear that our 
analysis is not nationally representative of jtpa training. Additionally, we 
cite many flaws associated with the design and implementation of the 
original njs that limit our analysis. However, no evidence exists to suggest 
that job training funded by jtpa and administered at the state and local 
level has changed so dramatically since 1989 that our findings are not 
relevant to the current program. 
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